As Linux-VServer is currently asleep, I thought about what may be possible only with the functionality the upstream Linux kernel provides and a helper module.

Today I got another broken DVD. It is the special edition of The Shawshank Redemption. It is rather new and libdvdread just chokes on it. At least the old one does, while the version in sid is able to read them, thanks to a small patch. The DVD exploits a feature of the standard. The filesystem on it (UDF) provides a size for the info files (VTS_XX_0.IFO) and the corresponding backup file (VTS_XX_0.BUP). This sizes are void, only the offsets within the files matters. The sizes are just too small and libdvdread checked that while reading them. Thanks to the patch, libdvdread now updates its internal knowledge while parsing the file. With the fix it is possible to play the DVD, at least with mplayer. But I want to play it on my machine without optical drive. I used dvdbackup to read other DVDs but it simply failed this time. This DVD also have broken backup files. They have different sizes, sometimes they even look empty, and contents than the real ones. With the corrent amount of force (just ignoring them) and some magic (#467075) dvdbackup copies something to disk which is playable with mplayer (libdvdread) and totem (libdvdnav). Okay, nothing is complete. This DVD contains another title set which seems to be never referenced. It have a completely bogus info file and a 1GB menu VOB. As I have a copy now, I'm away, watching a really great movie.

Xen support finally landed in upstream Linux. Okay, it is rather limited yet, but usable. It supports the following:

• Unprivileged domain (domU).
• x86_32 (i386 without PAE) and x86_32p (i386 with PAE).
• Console.
• netfront and blkfront.

Changes to the old Xen patch:

• Block devices does not support takeover of hdXY and sdXY. Use xvcX (xvca, xvcb, ...) as device names.
• Console is hvc0. You must supply console=hvc0 at the command line. (Use the extra definition in the Xen domain config file.)

Yet missing things:

• Suspend, resume (also makes migration impossible).
• Ballooning.
• x86_64. Xen upstream currently waits for an unified x86 tree.
• Privileged domain (dom0).
• netback, blkback, pciback and pcifront.

Hmm. Debian fixed this problems, but Ubuntu manages to get them also:

P: Configuring package volumeid
O: cp:
O: cannot stat  /etc/fstab'
O: : No such file or directory
O:
O: dpkg: error processing volumeid (--configure):
O:  subprocess post-installation script returned error exit status 1

I finaly got some time to do development. I decided to use this for extending cdebootstrap. And it got several new features and cleanups. Most notable change is the support to check Release files against the pgp-signature. This feature is enabled by default and uses the keyring supplied by debian-archive-keyring. It now also includes definitions for newer Ubuntu releases and it works in my testcases. The keyring needs to be supplied manualy. The rest are usual cleanups on the codebase and deprecation of some not longer used features.

After another debugging session I have to declare that caching of ressources with Vary headers is not working with squid 2.6. I have to check if this was working with squid 2.5 as Plone provides a predefined setup for this. Lets see what upstream say about this.

The linux kernel includes a SCSI target infrastructure since 2.6.20. Most of the code is located in the userspace and supports iSCSI, ibm i/pSeries virtual SCSI and Xen SCSIback. To work properly it needs a bunch of patches on top of 2.6.21-rc. After I got it working I did some tests with bonnie. First the target:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                 2G           32124  17 18063   6           45815   4 324.6   0

Now with the vSCSI initiator:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                 2G           18513   9 16208   5           34003   3 221.4   0

And the iSCSI initiator:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
                 2G           31631  14 12809   4           39241   5 306.3   1

As this tests was done on one OpenPower machine, all transfers uses DMA between the two systems. The vscsi case is currently limited to 128KiB per request, maybe this is a problem.

I thought the problems with pvmove was fixed with devmapper 1.02.12, but I just run into this problem again. Not sure if there was different fixes in later versions. pvmove suspends the devices and fails to reload tables under some conditions. This time it just blocked while the devices for /usr und /var was suspended. This means the system is dead. Update: It seems to be related to the number of LVs to move. Only one works all the time.

It happened again. The linux-2.6 2.6.20-1 release failed for arches which we don't build snapshots for. This means that none of them was ever built in the time after I commited the whole stuff. As this happens over and over again I consider this a real problem now. So the following arches needs new debian-kernel maintainers: alpha, hppa, and mips.

For etch, we will have apt security in place, so we can be sure that the stuff comes from the correct archive. But it is not possible to disable that checks only for one source, just for anything. Buildds uses at least one mirror: incoming.debian.org aka ftp-master.debian.org. There are two queues, the accepted autobuild queue and the main archive. The accepted autobuild queue is not signed at all, it does not provide a Release file. The archive needs some time to generate the Packages files each dinstall run and have broken sigs during this time. This means: buildds can't use APT security at all. And no, there is no other mechanism to ensure data integrity.

The buildd admin job is a rather dumb one. You get between 20 and 60 mails per day; most of them build logs. This logs, which are mostly sent unsecured through the public internet, have to be signed and the only available key is the personal key of the admin. So on one hand you have to make sure that the key is secure, on the other hand you have to find a way to sign a rather large amount of stuff. For the debian-kernel archive, which I operate, and the pkg-voip/pkg-gnome/pkg-kde-extras archive, which is operated by Kilian, we decided to sign the uploads automaticaly. Each buildd get its own key and the used DAK includes a patch which restricts this keys to do uploads of only binaries of the correct arches. This drasticaly reduces the time until a new package is uploaded; this means much less failed builds because a build dep is not yet built. Also it reduces the places where it is possible to do harm; you have to attack the buildd machine itself instead of the complete mail setup between buildd and admin.

Why is there no automatic or at least semi automatic infrastructure for LSB tests? The last published result is from the beginning of the last year and only for sarge.

The upcomming linux-2.6 release will introduce some changes in the Xen support.

Yes Junichi, you are correct, the behaviour of /proc/1/ cwd,root changed in 2.6.18. The relevant commit is 778c1144771f0064b6f51bee865cceb0d996f2f9. The behaviour is now really weird. readlink returns /, but in reality it points to somewhere else. Also it breaks the chroot detection.

I just take a look into the opcodes list of binutils for s390 and found the following entries:

b9b1 cu24 RRF_M0RR "convert utf-16 to utf-32" z9-109 zarch
b2a6 cu21 RRF_M0RR "convert utf-16 to utf-8" z9-109 zarch
b2a6 cuutf RRF_M0RR "convert unicode to utf-8" z9-109 zarch
b9b3 cu42 RRF_M0RR "convert utf-32 to utf-16" z9-109 zarch
b9b2 cu41 RRF_M0RR "convert utf-32 to utf-8" z9-109 zarch
b2a7 cu12 RRF_M0RR "convert utf-8 to utf-16" z9-109 zarch
b2a7 cutfu RRF_M0RR "convert utf-8 to unicode" z9-109 zarch
b9b0 cu14 RRF_M0RR "convert utf-8 to utf-32" z9-109 zarch

Since z9-109, it have own opcodes to convert between the different types of unicode.

It is done. The new Xen infrastructure is finished, although it is not yet completely in unstable. The main difference to the old one, the hypervisor and utils packages includes the complete version and an abi spec, just like our linux kernels. Also it is now possible to install more than one version of the utils at the same time. The userspace tries to find the correct tools for the running hypervisor. The following packages are provided now:

• xen-hypervisor-3.0.2-1-$flavour
• xen-utils-3.0.2-1
• xen-ioemu-3.0

and from the kernel

• xen-linux-system-$kernelversion

The only missing thing is a selection of the correct hypervisor in update-grub.

I did some performance tests with the T2000. The CPU is too slow to beat any current one from IBM (Power), AMD or Intel. My first test was building kernels. With -j32, a complete build needs about 9 minutes. For comparison, a 4-way UltraSparc 2 with 450MHz needs 30 minutes. So each core compares to an UltraSparc 2 with 700MHz. This numbers are not that phenomenal, so I added another test with ccache, to realy use the io system. With empty cache, it needs 10 minutes, with filled cache it needs 2.5 minutes. As application server, it is not that usable; they are cpu bound. Especially zope, which is known to not scale well on more cpus because of the python global interpreter lock, is not able to use this. A comparision with an Opteron 242 with 1.6GHz shows a factor of 5 in the computing power on virtual cpu.

Some people already know it, I got a new machine to play with. It is a Sparc running Debian etch:

# uname -a
Linux zee 2.6.17-2-vserver-sparc64 #1 SMP Sat Aug 26 12:28:58 UTC 2006 sparc64 GNU/Linux

The kernel is not yet available in Debian, it is handbuilt. But I hope that this will change soon. It is a small machine with some CPUs:

# grep ncpus /proc/cpuinfo
ncpus probed    : 32
ncpus active    : 32

It only have a little bit of RAM:

# grep MemTotal /proc/meminfo 
MemTotal:     33257432 kB

And really too less disk space:

SCSI device sda: 143374738 512-byte hdwr sectors (73408 MB)
SCSI device sdb: 143374738 512-byte hdwr sectors (73408 MB)

Yes, it is a T2000 from Sun with 8 real CPU cores:

cpu             : UltraSparc T1 (Niagara)
fpu             : UltraSparc T1 integrated FPU
prom            : OBP 4.19.0 2005/10/27 17:24
type            : sun4v

Each of the 8 cores can run 4 threads at the same time, Sun calls this CoolThreads, and Linux sees this as virtual CPUs.

Yes, the xen images from linux 2.6.16-18 in unstable does not work with the currect xen 3.0 in unstable and testing. The interface for the priviledged domain changed in an incompatible way. I'm working on that problem and 2.6.17-7 will contain a complete new infrastructure.

Some people may have noticed that a thread has recently started about Xen on debian-devel. To sum up the whole story, here it is. Guido Trotter and myself asked Adam Heath, the current official maintainer of Xen, if it was possible to help him (bug #342249) to package Xen 3.
He did not answer to our messages, as he seems to be MIA, so we started with Jeremy Bouse, Ralph Passgang and Yvette Chanco to work and we created a project on Alioth in order to package the latest release. We started our development from the package Ralph made previously. We don't know really why, but Bastian Blank, from the Debian kernel team, uploaded his own package of Xen 3, ignoring our request to work on our side, arguing that this was the work of kernel team to maintain Xen. It seems that actually, the kernel team only pretend to maintain the Dom0 and DomU kernel images, probably as soon as Xen in included in the vanilla kernel, and that Bastian took the decision alone to maintain the hypervizor and userspace tools. Today, we have functionnal and splitted packages of Xen 3, available on the Subversion repository. I made Xen packages with the latest development version of our packages and the Xen testing version. They are available from:

deb http://naquadah.org/~jd/debian/xen stable main

Official backports will be uploaded to backports.org as soon as we will have uploaded a version to sid. But for now, we have to cancel or bypass the upload Bastian made...